Skip to content

feat: functional trusted VSI binding, ladder signing and ownership hardening - #4

Draft
mstattma wants to merge 38 commits into
feat/live-video-vsifrom
feat/trusted-vsi-functional
Draft

mstattma wants to merge 38 commits into
feat/live-video-vsifrom
feat/trusted-vsi-functional

Conversation

@mstattma

@mstattma mstattma commented Sep 30, 2026 •

Copy link
Copy Markdown

Functional trusted VSI Python binding plus ladder/fragmented signing and ownership hardening, paired with castlabs/c2pa-rs feat/trusted-vsi-functional.

Current Qualified Pair

Python 97c820efde96af75c900c02e05457aab1ba2e1e1, native a6d4cdcc05638ee8dd0afce7fa5c850d7031a80c.

  • TrustedVsiSession and helpers admit only exact revision 3, native SDK token c2pa-rs/0.92.0-dev, required symbols and mask 63, before operational ctypes binding. No version range, fallback or trial operation. Contract: docs/trusted-vsi-python-contract.md.
  • Callback-triggered close is immediately logical; new admissions are refused and physical native teardown waits for admitted signing calls. Callback pins survive native free, including borrowed Signer/Context lifetimes. This is not generic thread safety: native operations still require serialization.
  • blocked is exposed by ctypes/status, with offset 18 and total native status size 24. Failure/recovery tests exercise it.
  • Stock-native scripted dynamic-assertion cases use has_dynamic_assertions(); source and installed-wheel paired jobs actually execute their functional checks.
  • The fragmented wrapper preserves the existing six-argument glob-aware ABI, supporting one shared manifest across segmented renditions without per-rendition fallback.
  • Ladder cleanup guidance permits deletion only of positively owned newly created outputs; sources and pre-existing destinations are never cleanup targets.

Merge Provenance

True merges retain ladder candidate 502b8bb2, generic ownership 7ba8615, reviewed follow-up a303db8 and callback-lifetime review fixes, without temporary CI-only commits or rewritten history. Ladder DynamicAssertion exceptions preserve identity and claim-signer interrupts propagate. Consume-first registry rejection parsing and opaque stream ownership remain anchored to rejected handle IDs.

Unreleased source identity 0.37.13.dev0 and immutable dev5 release inputs/locks/tooling remain distinct. Exact revision/version gating is intentionally coordinated, not source or binary authentication.

Current Qualification

  • Exact pair Linux/Windows non-publishing qualification: 37688529697. Each platform passed 286 focused, 831 non-threaded, 54 threaded and 286 installed-wheel tests, plus the real native ladder harness.
  • Local source qualification: 831 tests and 135 subtests; threaded 54; installed 286 and 18 subtests; installed release smoke 5. Review, stock-native and artifact provenance is in docs/archive/trusted-vsi-review-verification.md.
  • Required downstream stack: 37807071502, passing required Linux/Windows scopes for both CBOR versions. Informational full Windows retains 21 failures per version.
  • Frozen signer 0bf8949f passed fresh exact-source public/operational qualification with the downloaded native artifact: 84 passed, zero skips/failures per version. Execution evidence and selected integration review.

Historical Qualification And Limits

The old 12d265db/native 6b506352 pair (36808706395) is historical, superseded above. Stock 0.91.0 Windows x64 ownership/unit/ladder/threaded proof remains separate (36775608167); stock Windows ARM64 ownership was outside that scope. Ordinary stock-native hosted release jobs did not run in the current paired lane.

Remaining generic opaque-native concurrency/sticky-slot work stays in the roadmap. No generic thread-safety, native composed event-ID carry, production provider or universal platform guarantee is claimed.

No publication; dev5 tooling still refuses this source. Immutable historical release artifacts are unchanged.

tmathern and others added 30 commits August 27, 2026 15:38
* chore: Update c2pa version to v0.90.16

* Bump version from 0.37.8 to 0.37.9

* Bump version from 0.37.8 to 0.37.9
…uth#320)

* build: drop wheel, setuptools and pytest from runtime dependencies

None of the three is imported anywhere under `src/`. `c2pa.py` and `lib.py`
import only the standard library; `build.py` — the `download-artifacts` console
script — imports `requests` and, lazily, `toml`. Those two stay.

They are also already classified correctly elsewhere in the repo:

* `[build-system] requires` already lists `setuptools>=68.0.0` and `wheel`, so
  the build has what it needs and the runtime entries are duplicates.
* `requirements-dev.txt` lists `wheel` and `setuptools` under
  "# Build dependencies" and `pytest` under "# Testing dependencies".
* `.github/workflows/build.yml` installs pytest explicitly (`pip install
  pytest`, lines 285 and 377), so CI does not rely on the runtime declaration
  either.

Removing them is therefore a no-op for this repo's own build and test paths,
and it keeps three packages out of every consumer's production environment.

* build: declare pytest in a PEP 735 dev dependency group

Dropping pytest from `[project.dependencies]` left it undeclared in
pyproject.toml entirely, with `requirements-dev.txt` as the only manifest
naming it. `[dependency-groups] dev` states it where it belongs: installed
for contributors (`uv sync`, `pip install --group dev`) and, unlike
`[project.optional-dependencies]`, absent from the published package
metadata — which is the separation this branch is about.

The bound matches requirements-dev.txt (`pytest>=8.1.0`) rather than the
`>=7.4.0` the runtime entry carried; nothing installs the old one.

The comment above the remaining dependencies goes with it. The rationale
for keeping `toml` and `requests` belongs in the pull request, not in a
manifest that has carried no comments so far.

* build: keep pytest declared in requirements-dev.txt only

Review feedback: the PEP 735 group restated a bound that
`requirements-dev.txt` already carries, so the two could drift — which is
what this branch set out to stop, not to reproduce one line further down.

Nothing in the repo would have read the group. The Makefile's `install-deps`
and every workflow that installs dependencies do so with `pip install -r
requirements-dev.txt` (`build.yml` lines 52, 90/93, 164/167, 490/492), and
the wheel test jobs install pytest by name. The group was a declaration with
no consumer.

The published metadata — the point of this branch — is unaffected either
way, and the diff is now purely subtractive.
* Update c2pa version from v0.90.16 to v0.90.19

* Bump version from 0.37.9 to 0.37.10

* Bump version from 0.37.9 to 0.37.10
* chore: Update C2PA version to 0.90.22

* Bump version from 0.37.10 to 0.37.11

* Bump version from 0.37.10 to 0.37.11
…t-tooling

Add reusable c2pa-rs RC-preflight workflow; fix latent test bugs
…#327)

test_sdk_version compares the loaded native library's version against
c2pa-native-version.txt, but an RC-preflight run (per
test-c2pa-rs-source-build.yml) actually builds from whatever ref is in
c2pa-rs-preflight-ref.txt instead, so the test always failed on that
one assertion during a preflight even when everything else passed.

parse_native_version() now prefers c2pa-rs-preflight-ref.txt when
present, falling back to c2pa-native-version.txt otherwise -- the same
precedence the workflow itself uses to decide what to build. Verified
both paths locally: green against the pinned 0.90.22 with no preflight
file, and green against a c2pa-rc-v0.91.0-rc.3 build with the file
present.

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
contentauth#328)

contentauth#327's parse_native_version() fix preferred c2pa-rs-preflight-ref.txt
whenever it existed in the checked-out tree. That broke the *real*
build.yml jobs on contentauth#325: that PR adds the ref file to the branch, so an
ordinary tests-unix/tests-windows run (which downloads and installs
the actual pinned release, unrelated to the preflight workflow) picked
up the file too and wrongly expected the RC's version string, failing
with e.g. "'0.91.0-rc.3' not found in '0.90.22'".

The file's mere presence was never a reliable signal -- only
test-c2pa-rs-source-build.yml's own "Run tests" step actually builds
from that ref. Gate on a new C2PA_PREFLIGHT_RUN env var that only that
step sets instead.

Verified locally: with the pinned 0.90.22 installed and no env var
set, all 444 tests pass regardless of whether
c2pa-rs-preflight-ref.txt happens to exist in the tree.

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Fix test_sdk_version regression: gate on an env var, not file presence

contentauth#327's parse_native_version() fix preferred c2pa-rs-preflight-ref.txt
whenever it existed in the checked-out tree. That broke the *real*
build.yml jobs on contentauth#325: that PR adds the ref file to the branch, so an
ordinary tests-unix/tests-windows run (which downloads and installs
the actual pinned release, unrelated to the preflight workflow) picked
up the file too and wrongly expected the RC's version string, failing
with e.g. "'0.91.0-rc.3' not found in '0.90.22'".

The file's mere presence was never a reliable signal -- only
test-c2pa-rs-source-build.yml's own "Run tests" step actually builds
from that ref. Gate on a new C2PA_PREFLIGHT_RUN env var that only that
step sets instead.

Verified locally: with the pinned 0.90.22 installed and no env var
set, all 444 tests pass regardless of whether
c2pa-rs-preflight-ref.txt happens to exist in the tree.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* Point the RC preflight at c2pa-rs 0.91.0-rc.3

This is the entire diff this PR now carries on top of contentauth#328: pin the
preflight workflow at c2pa-rc-v0.91.0-rc.3 so it builds from that git
tag (no crates.io publish, no prebuilt GitHub release binaries for an
RC) and runs the full unit test suite against it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* Point the RC preflight at c2pa-rs 0.91.0-rc.3

This is the entire diff this PR now carries on top of contentauth#328: pin the
preflight workflow at c2pa-rc-v0.91.0-rc.3 so it builds from that git
tag (no crates.io publish, no prebuilt GitHub release binaries for an
RC) and runs the full unit test suite against it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* Preflight against c2pa-rs 0.91.0-rc.4

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* chore: bump c2pa-rs to v0.91.0

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* chore: Bump version from 0.37.11 to 0.37.12

* Bump version from 0.37.11 to 0.37.12
Register the native ladder export as optional, validate path arrays, preserve typed errors and allocation ownership, and use the modern single-sign builder lifecycle. Add focused binding tests and isolated stock/candidate native qualification lanes without changing release pins.

Co-authored-by: bibinbaby444 <bibinbaby444@gmail.com>

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Use c2pa_free for new manifest allocations, include focused ladder tests in existing CI runs, reject optimized verification harness execution, and document the modern builder lifecycle.
Include a documented synthetic fixture and offline native smoke in the existing CI selection. Enforce candidate capability on request, verify the native signature and shared manifest, and cover typed ctypes conversion without changing stock native pins or builder semantics.

Co-authored-by: bibinbaby444 <bibinbaby444@gmail.com>
…idation

# Conflicts:
#	docs/class-diagram.md
#	pyproject.toml
#	src/c2pa/c2pa.py
#	tests/test_unit_tests.py
…o 203dc08d

True merge of fix/native-resource-ownership-upstream 7ba8615. Takes its
anchored registry-tag parsing (strip one 'Other: ' prefix, match at start)
so tags quoted inside another error's payload no longer establish
ownership, its _handle_value and consume-first triage, its C2paStream
docstring, generic ownership tests and ownership documentation, keeping the
consolidation's fork content. Opaque-registry-only consumed-id checks move
to tests/test_native_ownership_opaque.py.

Paired CI now builds castlabs/c2pa-rs 203dc08d (ContentAuth main 69907b5a
merged; 0.92.0-dev, Rust 1.96.0 unchanged).
… of skipping

Require the opaque registry (odd object ids) with a fixture that fails when
C2PA_TRUSTED_VSI_ABI_REQUIRED/FUNCTIONAL_REQUIRED is set, list the new file
in the installed-wheel contract, and mark the 203dc08d pairing qualification
as pending in the contract status.
@mstattma

Copy link
Copy Markdown
Author

@BibinBaby444 requesting your review (a formal review request isn't possible: you're not a collaborator on this repository). Companion draft PRs: castlabs/c2pa-rs#15, #4, castlabs/stardustproof-keystore#13, castlabs/stardustproof-c2pa-signer#25.

@mstattma

mstattma commented Oct 1, 2026

Copy link
Copy Markdown
Author

The independently reviewed ownership follow-up a303db8 is true-merged (not cherry-picked) into functional head 5c64f2cc090eeb29506bc766faa69b959e4ed982; the CI-only commit remains excluded. The remaining native-registry checks and stock Windows ARM64 ownership qualification are in docs/roadmap.md. Paired run 36793704783 passed on Linux and Windows against native 203dc08d: 186 focused, real-native ladder harness, 730 non-threaded, 54 threaded, and 186 installed-wheel tests per platform. Stock 0.91.0 Windows x64 passed in mstattma run 36775608167. @BibinBaby444 please review the new merge abef446 and roadmap commits bc9e99d, 5c64f2c. No release or dev5 evidence was changed.

@BibinBaby444 BibinBaby444 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes requested. One memory-safety bug reachable from plain Python, plus one native state that's invisible to Python. Verified at 12d265db against a native built from c2pa-rs d738fbb2 (debug, unstable_live_video). Note that CI is not evidence here yet: every test job on this PR was skipped, including "Trusted VSI paired-source API"; only format and tooling checks ran.

P1: closing a session during a native call frees callbacks native is still using.

  • TrustedVsiSession._release drops _signer_callback_cb, _dynamic_assertion_cbs and _trusted_vsi_callback, and _teardown calls it before c2pa_free.
  • Native free defers the actual drop while a call holds the borrow, so the in-flight finalize_init_uuid keeps calling through the freed thunks.
  • Once the Context has been closed (which the contract allows), the session holds the only reference to the claim-signer thunk.
  • A single thread is enough. Close the session from inside its own signer_binding callback, then allocate fresh SignerCallbacks:
    finalize raised: _C2paSignature('Signature: internal error (bad parameter: signer callback returned error code -1)')
    real claim callback calls: []  impostor calls: [1876]
    
  • Native jumped through the freed claim thunk into an unrelated callback, which received the 1,876-byte claim payload. With different reuse that's a crash or a call into arbitrary code.
  • Suggest keeping the callback pins alive for the object's lifetime (don't clear them in _release), or deferring close() until in-flight calls drain.
  • The same drop-on-close pattern exists in the older live-video VSI session and Builder _releases, and the native contract text ("until the handle is freed") should say "until the deferred drop".

P2: blocked is never exposed.

  • Native C2paLiveVideoTrustedVsiStatusV1 has blocked: bool between has_exhaustion_reason and exhaustion_reason, at offset 18.
  • The ctypes struct, TrustedVsiStatus and status() all omit it. The layout still lines up, since Python treats the byte as padding, so there's no memory hazard.
  • But Python can't see the one state the contract tells adapters to act on ("After a failure once an external signing call began, the session is blocked. Discard it, construct a NEW session…"):
    before failure: native blocked byte[18] = 0  -> status() has no blocked field
    after failure:  native blocked byte[18] = 1  -> status() unchanged
    next call: _C2paOther Other: bad parameter: session is blocked after a failed external signing step; …
    
  • Add the field and a blocked attribute on TrustedVsiStatus, update the offsets pinned in tests/test_trusted_vsi_api.py, and add a paired test that asserts status().blocked after a callback failure.

P2: the stock-native lane will fail, and CI hasn't run it. The dynamic variant of test_scripted_wrappers_store_base_exceptions_return_minus_one_and_reraise calls signer.add_dynamic_assertion unguarded. The -k "not paired" lane in build.yml downloads upstream c2pa-v0.91.0 (c2pa-native-version.txt), and v0.91.0 (e7cc666c) doesn't export c2pa_signer_add_dynamic_assertion, so those four parametrizations raise NotSupported. Guard the variant on has_dynamic_assertions() (or stub the export in _ScriptedNative), then get the skipped jobs to actually run.

P3

  • Ladder error guidance drops a guarantee. docs/ladder-signing.md and the sign_ladder docstring say "Errors may leave partial newly created outputs; discard these files", but leave out native's guarantee that "Existing files, including every source, are never overwritten". The native error doesn't say which path failed, so say callers should delete only destinations they confirmed absent before the call. Otherwise a cleanup-everything handler deletes a pre-existing file.
  • The version gate is a release label, not an ABI identity. 0.92.0-dev is upstream's workspace version, so the gate effectively rests on the symbol plus mask 63. A native ABI-revision probe would make a future in-place V1 struct change detectable.

Defer physical teardown until admitted native calls drain, preserve callback pins through free, and expose trusted VSI blocked status. Cover reentrant close and stock-native behavior, adapt paired tests to the hardened native contract, and clarify output cleanup and qualification holds.
@mstattma

mstattma commented Oct 8, 2026

Copy link
Copy Markdown
Author

Resolution mapping for the existing review at Python 97c820efde96af75c900c02e05457aab1ba2e1e1, paired with native a6d4cdcc:

  • P1 reentrant-close callback lifetime: close becomes logical immediately, refuses new admissions, and defers physical native teardown until admitted calls drain. Callback pins remain alive through native destruction, including borrowed Signer/Context lifetimes. Native operations still require serialization; generic thread safety is not claimed.
  • P2 blocked state: ctypes, the status dataclass and status() expose blocked, at native offset 18 with total status size 24; real failure/recovery coverage is included.
  • P2 stock-native dynamic cases: scripted dynamic-assertion cases check has_dynamic_assertions(). Paired jobs now actually run focused, ladder, non-threaded, threaded and installed-wheel checks. Stock 0.91.0 and Windows ARM64 limits remain separate, not inferred from this pairing.
  • P3 unsafe ladder cleanup guidance: documentation forbids deletion of sources or pre-existing destinations; only positively owned new outputs are cleanup candidates.
  • P3 label-only compatibility gate: exact revision 3, SDK token, required symbols and capability mask 63 are checked before trusted operational ctypes binding. Incompatible native libraries fail closed.

The integrated fragmented wrapper also restores the existing six-argument glob-aware ABI and shared-manifest segmented signing, without per-rendition fallback or ownership changes.

Exact-head paired qualification 37688529697 passed on Linux/Windows: 286 focused, 831 non-threaded, 54 threaded and 286 installed-wheel tests per platform, plus the real-native ladder harness. Source/stock/installed evidence is in docs/archive/trusted-vsi-review-verification.md. Required downstream qualification and fresh frozen-source execution are recorded separately with their exact heads and limitations.

Ordinary stock-native hosted release jobs did not run in this paired lane. Immutable dev5 artifacts, release inputs and publication gates are unchanged; no release or universal platform guarantee is claimed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants